{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YajxPiJCuy5j"
      },
      "source": [
        "##### Copyright 2020 Google LLC.\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7mLBFZ-uu-Bh"
      },
      "outputs": [],
      "source": [
        "# Copyright 2022 The Google Research Authors.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9CimtUIDwna4"
      },
      "source": [
        "# Human evaluation of visual metrics\n",
        "\n",
        "This colab explores correlations between the mucped22 dataset and various objective visual metrics.\n",
        "\n",
        "Unlike many other datasets, these evaluations:\n",
        " *   Are made using only compression distortions (since the distortions were created using AVIF, JPEG, and JXL encoders), which will benefit metrics designed for compression artifacts.\n",
        " *   Are made by human evaluators experienced with image quality comparisons, which will benefit smaller distortions, relatively speaking, since unexperienced evaluators often don't notice them.\n",
        " *   Are made using two-alternative-forced-choice with a reference image instead of MOS, which will reduce the noise, since evaluators don't need to calibrate their MOS scores.\n",
        "\n",
        "For each image, each distortion (method and quality setting) has then been ranked using ELO, to provide an expected human-rated ranking for each distortion.\n",
        "\n",
        "This ranking will allow a comparison of the various metrics across different levels of distortion, e.g. near just-noticeable-differences vs far from just-noticeable-differences.\n",
        "\n",
        " "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mVjmAAMuLszT",
        "outputId": "6c5207c9-0da0-4896-dc66-fcc00af78e37"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting pandas_bokeh\n",
            "  Downloading pandas_bokeh-0.5.5-py2.py3-none-any.whl (29 kB)\n",
            "Requirement already satisfied: bokeh\u003e=2.0 in /usr/local/lib/python3.7/dist-packages (from pandas_bokeh) (2.3.3)\n",
            "Requirement already satisfied: pandas\u003e=0.22.0 in /usr/local/lib/python3.7/dist-packages (from pandas_bokeh) (1.3.5)\n",
            "Requirement already satisfied: numpy\u003e=1.11.3 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (1.21.5)\n",
            "Requirement already satisfied: Jinja2\u003e=2.9 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (2.11.3)\n",
            "Requirement already satisfied: python-dateutil\u003e=2.1 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (2.8.2)\n",
            "Requirement already satisfied: tornado\u003e=5.1 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (5.1.1)\n",
            "Requirement already satisfied: PyYAML\u003e=3.10 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (3.13)\n",
            "Requirement already satisfied: pillow\u003e=7.1.0 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (7.1.2)\n",
            "Requirement already satisfied: typing-extensions\u003e=3.7.4 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (3.10.0.2)\n",
            "Requirement already satisfied: packaging\u003e=16.8 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (21.3)\n",
            "Requirement already satisfied: MarkupSafe\u003e=0.23 in /usr/local/lib/python3.7/dist-packages (from Jinja2\u003e=2.9-\u003ebokeh\u003e=2.0-\u003epandas_bokeh) (2.0.1)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,\u003e=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging\u003e=16.8-\u003ebokeh\u003e=2.0-\u003epandas_bokeh) (3.0.7)\n",
            "Requirement already satisfied: pytz\u003e=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas\u003e=0.22.0-\u003epandas_bokeh) (2018.9)\n",
            "Requirement already satisfied: six\u003e=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil\u003e=2.1-\u003ebokeh\u003e=2.0-\u003epandas_bokeh) (1.15.0)\n",
            "Installing collected packages: pandas-bokeh\n",
            "Successfully installed pandas-bokeh-0.5.5\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import functools\n",
        "import bokeh.io\n",
        "!pip install pandas_bokeh\n",
        "import pandas_bokeh\n",
        "import requests\n",
        "import json\n",
        "import numpy as np\n",
        "bokeh.io.output_notebook()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1jPN8OpOy0Kn"
      },
      "source": [
        "First download the dataset containing all evaluations."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "N4R1dG01sVEJ"
      },
      "outputs": [],
      "source": [
        "!wget --quiet --no-check-certificate https://storage.googleapis.com/gresearch/mucped22/evaluations.json"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "osV-uT-Kh4HK"
      },
      "outputs": [],
      "source": [
        "with open('evaluations.json') as f:\n",
        "  data = pd.DataFrame(json.load(f))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "eGIkAdlWo-di",
        "outputId": "b21eefd6-cff2-444a-ae08-ea5556e78e19"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  \u003cdiv id=\"df-9c5949dd-e0f8-42d9-8efe-46c4aa30bb32\"\u003e\n",
              "    \u003cdiv class=\"colab-df-container\"\u003e\n",
              "      \u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003ecrop\u003c/th\u003e\n",
              "      \u003cth\u003egreater\u003c/th\u003e\n",
              "      \u003cth\u003eimage\u003c/th\u003e\n",
              "      \u003cth\u003elesser\u003c/th\u003e\n",
              "      \u003cth\u003erandom_choice\u003c/th\u003e\n",
              "      \u003cth\u003erater_time_ms\u003c/th\u003e\n",
              "      \u003cth\u003eimage_dims\u003c/th\u003e\n",
              "      \u003cth\u003egreater_butteraugli_max\u003c/th\u003e\n",
              "      \u003cth\u003egreater_butteraugli_6\u003c/th\u003e\n",
              "      \u003cth\u003elesser_butteraugli_max\u003c/th\u003e\n",
              "      \u003cth\u003e...\u003c/th\u003e\n",
              "      \u003cth\u003elesser_fsim-rgb\u003c/th\u003e\n",
              "      \u003cth\u003egreater_msssim-y\u003c/th\u003e\n",
              "      \u003cth\u003elesser_msssim-y\u003c/th\u003e\n",
              "      \u003cth\u003egreater_nlpd-y\u003c/th\u003e\n",
              "      \u003cth\u003elesser_nlpd-y\u003c/th\u003e\n",
              "      \u003cth\u003egreater_ssimulacra\u003c/th\u003e\n",
              "      \u003cth\u003elesser_ssimulacra\u003c/th\u003e\n",
              "      \u003cth\u003egreater_elo\u003c/th\u003e\n",
              "      \u003cth\u003elesser_elo\u003c/th\u003e\n",
              "      \u003cth\u003erater_flips\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003e[176, 279, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_6\u003c/td\u003e\n",
              "      \u003ctd\u003eedgar-castrejon-J_NbCcGPBOw-unsplash.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_17\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "      \u003ctd\u003e75125\u003c/td\u003e\n",
              "      \u003ctd\u003e[1280, 1920]\u003c/td\u003e\n",
              "      \u003ctd\u003e6.792748\u003c/td\u003e\n",
              "      \u003ctd\u003e3.291877\u003c/td\u003e\n",
              "      \u003ctd\u003e2.287096\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998617\u003c/td\u003e\n",
              "      \u003ctd\u003e0.975003\u003c/td\u003e\n",
              "      \u003ctd\u003e0.993973\u003c/td\u003e\n",
              "      \u003ctd\u003e0.161716\u003c/td\u003e\n",
              "      \u003ctd\u003e0.078657\u003c/td\u003e\n",
              "      \u003ctd\u003e0.096484\u003c/td\u003e\n",
              "      \u003ctd\u003e0.045761\u003c/td\u003e\n",
              "      \u003ctd\u003e1537.137226\u003c/td\u003e\n",
              "      \u003ctd\u003e2172.213746\u003c/td\u003e\n",
              "      \u003ctd\u003e9\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003e[149, 32, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_6\u003c/td\u003e\n",
              "      \u003ctd\u003e4KK2_20150823_130650_314.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_11\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "      \u003ctd\u003e41883\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",
              "      \u003ctd\u003e6.830727\u003c/td\u003e\n",
              "      \u003ctd\u003e3.382573\u003c/td\u003e\n",
              "      \u003ctd\u003e1.990106\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998584\u003c/td\u003e\n",
              "      \u003ctd\u003e0.985169\u003c/td\u003e\n",
              "      \u003ctd\u003e0.997969\u003c/td\u003e\n",
              "      \u003ctd\u003e0.145104\u003c/td\u003e\n",
              "      \u003ctd\u003e0.050339\u003c/td\u003e\n",
              "      \u003ctd\u003e0.072111\u003c/td\u003e\n",
              "      \u003ctd\u003e0.019148\u003c/td\u003e\n",
              "      \u003ctd\u003e1972.598657\u003c/td\u003e\n",
              "      \u003ctd\u003e2291.390798\u003c/td\u003e\n",
              "      \u003ctd\u003e13\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003e[1089, 489, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_11\u003c/td\u003e\n",
              "      \u003ctd\u003e6G7M_20150404_121844_208.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_17\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "      \u003ctd\u003e48385\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",
              "      \u003ctd\u003e1.760315\u003c/td\u003e\n",
              "      \u003ctd\u003e0.828016\u003c/td\u003e\n",
              "      \u003ctd\u003e1.656251\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.999163\u003c/td\u003e\n",
              "      \u003ctd\u003e0.999303\u003c/td\u003e\n",
              "      \u003ctd\u003e0.999264\u003c/td\u003e\n",
              "      \u003ctd\u003e0.030033\u003c/td\u003e\n",
              "      \u003ctd\u003e0.030637\u003c/td\u003e\n",
              "      \u003ctd\u003e0.010733\u003c/td\u003e\n",
              "      \u003ctd\u003e0.011180\u003c/td\u003e\n",
              "      \u003ctd\u003e2268.583209\u003c/td\u003e\n",
              "      \u003ctd\u003e2309.359159\u003c/td\u003e\n",
              "      \u003ctd\u003e4\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003e[433, 286, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_14\u003c/td\u003e\n",
              "      \u003ctd\u003emichael-niessl-KraoHdRYrRE-unsplash.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_11\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e36774\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1080]\u003c/td\u003e\n",
              "      \u003ctd\u003e4.110695\u003c/td\u003e\n",
              "      \u003ctd\u003e1.995982\u003c/td\u003e\n",
              "      \u003ctd\u003e2.432354\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998708\u003c/td\u003e\n",
              "      \u003ctd\u003e0.993285\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998220\u003c/td\u003e\n",
              "      \u003ctd\u003e0.071748\u003c/td\u003e\n",
              "      \u003ctd\u003e0.037420\u003c/td\u003e\n",
              "      \u003ctd\u003e0.043709\u003c/td\u003e\n",
              "      \u003ctd\u003e0.019095\u003c/td\u003e\n",
              "      \u003ctd\u003e1991.945557\u003c/td\u003e\n",
              "      \u003ctd\u003e2241.368691\u003c/td\u003e\n",
              "      \u003ctd\u003e10\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003e[4, 99, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_6\u003c/td\u003e\n",
              "      \u003ctd\u003e0127_20161022_144117_906.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_14\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e22468\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1440]\u003c/td\u003e\n",
              "      \u003ctd\u003e5.866006\u003c/td\u003e\n",
              "      \u003ctd\u003e2.645947\u003c/td\u003e\n",
              "      \u003ctd\u003e3.550395\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.994832\u003c/td\u003e\n",
              "      \u003ctd\u003e0.988954\u003c/td\u003e\n",
              "      \u003ctd\u003e0.994369\u003c/td\u003e\n",
              "      \u003ctd\u003e0.095547\u003c/td\u003e\n",
              "      \u003ctd\u003e0.069588\u003c/td\u003e\n",
              "      \u003ctd\u003e0.049872\u003c/td\u003e\n",
              "      \u003ctd\u003e0.035815\u003c/td\u003e\n",
              "      \u003ctd\u003e1851.898409\u003c/td\u003e\n",
              "      \u003ctd\u003e1945.269660\u003c/td\u003e\n",
              "      \u003ctd\u003e9\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e...\u003c/th\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e12978\u003c/th\u003e\n",
              "      \u003ctd\u003e[887, 59, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_23\u003c/td\u003e\n",
              "      \u003ctd\u003e6G7M_20150404_121844_208.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_40\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "      \u003ctd\u003e7983\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",
              "      \u003ctd\u003e1.877585\u003c/td\u003e\n",
              "      \u003ctd\u003e0.923585\u003c/td\u003e\n",
              "      \u003ctd\u003e2.143382\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.999173\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998298\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998512\u003c/td\u003e\n",
              "      \u003ctd\u003e0.051131\u003c/td\u003e\n",
              "      \u003ctd\u003e0.048955\u003c/td\u003e\n",
              "      \u003ctd\u003e0.020006\u003c/td\u003e\n",
              "      \u003ctd\u003e0.017322\u003c/td\u003e\n",
              "      \u003ctd\u003e2356.875932\u003c/td\u003e\n",
              "      \u003ctd\u003e2388.690716\u003c/td\u003e\n",
              "      \u003ctd\u003e6\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e12979\u003c/th\u003e\n",
              "      \u003ctd\u003e[1121, 175, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_32\u003c/td\u003e\n",
              "      \u003ctd\u003e47L8_20150518_144020_604.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_30\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e1702517\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",
              "      \u003ctd\u003e4.943964\u003c/td\u003e\n",
              "      \u003ctd\u003e2.460808\u003c/td\u003e\n",
              "      \u003ctd\u003e8.950997\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.969582\u003c/td\u003e\n",
              "      \u003ctd\u003e0.983741\u003c/td\u003e\n",
              "      \u003ctd\u003e0.955344\u003c/td\u003e\n",
              "      \u003ctd\u003e0.127071\u003c/td\u003e\n",
              "      \u003ctd\u003e0.201221\u003c/td\u003e\n",
              "      \u003ctd\u003e0.071460\u003c/td\u003e\n",
              "      \u003ctd\u003e0.155907\u003c/td\u003e\n",
              "      \u003ctd\u003e1937.790267\u003c/td\u003e\n",
              "      \u003ctd\u003e1402.672706\u003c/td\u003e\n",
              "      \u003ctd\u003e5\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e12980\u003c/th\u003e\n",
              "      \u003ctd\u003e[1121, 175, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_21\u003c/td\u003e\n",
              "      \u003ctd\u003e0006_20160721_220618_556.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_32\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e1430\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1440]\u003c/td\u003e\n",
              "      \u003ctd\u003e2.339564\u003c/td\u003e\n",
              "      \u003ctd\u003e1.123564\u003c/td\u003e\n",
              "      \u003ctd\u003e4.024082\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.997534\u003c/td\u003e\n",
              "      \u003ctd\u003e0.999049\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998433\u003c/td\u003e\n",
              "      \u003ctd\u003e0.029906\u003c/td\u003e\n",
              "      \u003ctd\u003e0.036302\u003c/td\u003e\n",
              "      \u003ctd\u003e0.011507\u003c/td\u003e\n",
              "      \u003ctd\u003e0.012152\u003c/td\u003e\n",
              "      \u003ctd\u003e2202.608359\u003c/td\u003e\n",
              "      \u003ctd\u003e1870.660112\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e12981\u003c/th\u003e\n",
              "      \u003ctd\u003e[84, 358, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_10\u003c/td\u003e\n",
              "      \u003ctd\u003ericardo-gomez-angel-2GglDxer-Ew-unsplash.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_32\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e5731\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1319]\u003c/td\u003e\n",
              "      \u003ctd\u003e3.816417\u003c/td\u003e\n",
              "      \u003ctd\u003e1.697080\u003c/td\u003e\n",
              "      \u003ctd\u003e5.742501\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.985294\u003c/td\u003e\n",
              "      \u003ctd\u003e0.991662\u003c/td\u003e\n",
              "      \u003ctd\u003e0.980832\u003c/td\u003e\n",
              "      \u003ctd\u003e0.105602\u003c/td\u003e\n",
              "      \u003ctd\u003e0.189351\u003c/td\u003e\n",
              "      \u003ctd\u003e0.048339\u003c/td\u003e\n",
              "      \u003ctd\u003e0.077644\u003c/td\u003e\n",
              "      \u003ctd\u003e2174.169512\u003c/td\u003e\n",
              "      \u003ctd\u003e2083.620449\u003c/td\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e12982\u003c/th\u003e\n",
              "      \u003ctd\u003e[84, 358, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_10\u003c/td\u003e\n",
              "      \u003ctd\u003e6G7M_20150403_174605_886.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_21\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e339\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",
              "      \u003ctd\u003e3.647443\u003c/td\u003e\n",
              "      \u003ctd\u003e1.543176\u003c/td\u003e\n",
              "      \u003ctd\u003e3.500668\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.996085\u003c/td\u003e\n",
              "      \u003ctd\u003e0.995072\u003c/td\u003e\n",
              "      \u003ctd\u003e0.991573\u003c/td\u003e\n",
              "      \u003ctd\u003e0.067868\u003c/td\u003e\n",
              "      \u003ctd\u003e0.094789\u003c/td\u003e\n",
              "      \u003ctd\u003e0.022608\u003c/td\u003e\n",
              "      \u003ctd\u003e0.036871\u003c/td\u003e\n",
              "      \u003ctd\u003e2235.728150\u003c/td\u003e\n",
              "      \u003ctd\u003e2219.418325\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003cp\u003e12983 rows × 24 columns\u003c/p\u003e\n",
              "\u003c/div\u003e\n",
              "      \u003cbutton class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9c5949dd-e0f8-42d9-8efe-46c4aa30bb32')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\"\u003e\n",
              "        \n",
              "  \u003csvg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\"\u003e\n",
              "    \u003cpath d=\"M0 0h24v24H0V0z\" fill=\"none\"/\u003e\n",
              "    \u003cpath d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/\u003e\u003cpath d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/\u003e\n",
              "  \u003c/svg\u003e\n",
              "      \u003c/button\u003e\n",
              "      \n",
              "  \u003cstyle\u003e\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  \u003c/style\u003e\n",
              "\n",
              "      \u003cscript\u003e\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-9c5949dd-e0f8-42d9-8efe-46c4aa30bb32 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-9c5949dd-e0f8-42d9-8efe-46c4aa30bb32');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '\u003ca target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb\u003edata table notebook\u003c/a\u003e'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      \u003c/script\u003e\n",
              "    \u003c/div\u003e\n",
              "  \u003c/div\u003e\n",
              "  "
            ],
            "text/plain": [
              "                        crop  greater  \\\n",
              "0       [176, 279, 768, 768]   dist_6   \n",
              "1        [149, 32, 768, 768]   dist_6   \n",
              "2      [1089, 489, 768, 768]  dist_11   \n",
              "3       [433, 286, 768, 768]  dist_14   \n",
              "4          [4, 99, 768, 768]   dist_6   \n",
              "...                      ...      ...   \n",
              "12978    [887, 59, 768, 768]  dist_23   \n",
              "12979  [1121, 175, 768, 768]  dist_32   \n",
              "12980  [1121, 175, 768, 768]  dist_21   \n",
              "12981    [84, 358, 768, 768]  dist_10   \n",
              "12982    [84, 358, 768, 768]  dist_10   \n",
              "\n",
              "                                              image   lesser  random_choice  \\\n",
              "0          edgar-castrejon-J_NbCcGPBOw-unsplash.png  dist_17          False   \n",
              "1                      4KK2_20150823_130650_314.png  dist_11          False   \n",
              "2                      6G7M_20150404_121844_208.png  dist_17          False   \n",
              "3           michael-niessl-KraoHdRYrRE-unsplash.png  dist_11           True   \n",
              "4                      0127_20161022_144117_906.png  dist_14           True   \n",
              "...                                             ...      ...            ...   \n",
              "12978                  6G7M_20150404_121844_208.png  dist_40          False   \n",
              "12979                  47L8_20150518_144020_604.png  dist_30           True   \n",
              "12980                  0006_20160721_220618_556.png  dist_32           True   \n",
              "12981  ricardo-gomez-angel-2GglDxer-Ew-unsplash.png  dist_32           True   \n",
              "12982                  6G7M_20150403_174605_886.png  dist_21           True   \n",
              "\n",
              "       rater_time_ms    image_dims  greater_butteraugli_max  \\\n",
              "0              75125  [1280, 1920]                 6.792748   \n",
              "1              41883  [1920, 1424]                 6.830727   \n",
              "2              48385  [1920, 1424]                 1.760315   \n",
              "3              36774  [1920, 1080]                 4.110695   \n",
              "4              22468  [1920, 1440]                 5.866006   \n",
              "...              ...           ...                      ...   \n",
              "12978           7983  [1920, 1424]                 1.877585   \n",
              "12979        1702517  [1920, 1424]                 4.943964   \n",
              "12980           1430  [1920, 1440]                 2.339564   \n",
              "12981           5731  [1920, 1319]                 3.816417   \n",
              "12982            339  [1920, 1424]                 3.647443   \n",
              "\n",
              "       greater_butteraugli_6  lesser_butteraugli_max  ...  lesser_fsim-rgb  \\\n",
              "0                   3.291877                2.287096  ...         0.998617   \n",
              "1                   3.382573                1.990106  ...         0.998584   \n",
              "2                   0.828016                1.656251  ...         0.999163   \n",
              "3                   1.995982                2.432354  ...         0.998708   \n",
              "4                   2.645947                3.550395  ...         0.994832   \n",
              "...                      ...                     ...  ...              ...   \n",
              "12978               0.923585                2.143382  ...         0.999173   \n",
              "12979               2.460808                8.950997  ...         0.969582   \n",
              "12980               1.123564                4.024082  ...         0.997534   \n",
              "12981               1.697080                5.742501  ...         0.985294   \n",
              "12982               1.543176                3.500668  ...         0.996085   \n",
              "\n",
              "       greater_msssim-y  lesser_msssim-y  greater_nlpd-y  lesser_nlpd-y  \\\n",
              "0              0.975003         0.993973        0.161716       0.078657   \n",
              "1              0.985169         0.997969        0.145104       0.050339   \n",
              "2              0.999303         0.999264        0.030033       0.030637   \n",
              "3              0.993285         0.998220        0.071748       0.037420   \n",
              "4              0.988954         0.994369        0.095547       0.069588   \n",
              "...                 ...              ...             ...            ...   \n",
              "12978          0.998298         0.998512        0.051131       0.048955   \n",
              "12979          0.983741         0.955344        0.127071       0.201221   \n",
              "12980          0.999049         0.998433        0.029906       0.036302   \n",
              "12981          0.991662         0.980832        0.105602       0.189351   \n",
              "12982          0.995072         0.991573        0.067868       0.094789   \n",
              "\n",
              "       greater_ssimulacra  lesser_ssimulacra  greater_elo   lesser_elo  \\\n",
              "0                0.096484           0.045761  1537.137226  2172.213746   \n",
              "1                0.072111           0.019148  1972.598657  2291.390798   \n",
              "2                0.010733           0.011180  2268.583209  2309.359159   \n",
              "3                0.043709           0.019095  1991.945557  2241.368691   \n",
              "4                0.049872           0.035815  1851.898409  1945.269660   \n",
              "...                   ...                ...          ...          ...   \n",
              "12978            0.020006           0.017322  2356.875932  2388.690716   \n",
              "12979            0.071460           0.155907  1937.790267  1402.672706   \n",
              "12980            0.011507           0.012152  2202.608359  1870.660112   \n",
              "12981            0.048339           0.077644  2174.169512  2083.620449   \n",
              "12982            0.022608           0.036871  2235.728150  2219.418325   \n",
              "\n",
              "       rater_flips  \n",
              "0                9  \n",
              "1               13  \n",
              "2                4  \n",
              "3               10  \n",
              "4                9  \n",
              "...            ...  \n",
              "12978            6  \n",
              "12979            5  \n",
              "12980            0  \n",
              "12981            1  \n",
              "12982            0  \n",
              "\n",
              "[12983 rows x 24 columns]"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iNn8ft-xy6zE"
      },
      "source": [
        "Then decorate it with whether the crop settings were actually compatible with the image size (a few, ~15, evaluations have this bug), and the worst ELO of both distortions.\n",
        "Finally filter out all evaluations where the evaluator didn't seem to do a good job (didn't flip between distortions more than 2 times, didn't spend more than 3 seconds on the evaluation)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "0sFK6hCgMl_9",
        "outputId": "a6603c4b-d55a-48a6-89cc-cb10a0324207"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  \u003cdiv id=\"df-73737e54-fd5b-44a9-a4dc-5e3c92c1719d\"\u003e\n",
              "    \u003cdiv class=\"colab-df-container\"\u003e\n",
              "      \u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003ecrop\u003c/th\u003e\n",
              "      \u003cth\u003egreater\u003c/th\u003e\n",
              "      \u003cth\u003eimage\u003c/th\u003e\n",
              "      \u003cth\u003elesser\u003c/th\u003e\n",
              "      \u003cth\u003erandom_choice\u003c/th\u003e\n",
              "      \u003cth\u003erater_time_ms\u003c/th\u003e\n",
              "      \u003cth\u003eimage_dims\u003c/th\u003e\n",
              "      \u003cth\u003egreater_butteraugli_max\u003c/th\u003e\n",
              "      \u003cth\u003egreater_butteraugli_6\u003c/th\u003e\n",
              "      \u003cth\u003elesser_butteraugli_max\u003c/th\u003e\n",
              "      \u003cth\u003e...\u003c/th\u003e\n",
              "      \u003cth\u003elesser_msssim-y\u003c/th\u003e\n",
              "      \u003cth\u003egreater_nlpd-y\u003c/th\u003e\n",
              "      \u003cth\u003elesser_nlpd-y\u003c/th\u003e\n",
              "      \u003cth\u003egreater_ssimulacra\u003c/th\u003e\n",
              "      \u003cth\u003elesser_ssimulacra\u003c/th\u003e\n",
              "      \u003cth\u003egreater_elo\u003c/th\u003e\n",
              "      \u003cth\u003elesser_elo\u003c/th\u003e\n",
              "      \u003cth\u003erater_flips\u003c/th\u003e\n",
              "      \u003cth\u003ecomplete_crop\u003c/th\u003e\n",
              "      \u003cth\u003eworst_elo\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003e[176, 279, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_6\u003c/td\u003e\n",
              "      \u003ctd\u003eedgar-castrejon-J_NbCcGPBOw-unsplash.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_17\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "      \u003ctd\u003e75125\u003c/td\u003e\n",
              "      \u003ctd\u003e[1280, 1920]\u003c/td\u003e\n",
              "      \u003ctd\u003e6.792748\u003c/td\u003e\n",
              "      \u003ctd\u003e3.291877\u003c/td\u003e\n",
              "      \u003ctd\u003e2.287096\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.993973\u003c/td\u003e\n",
              "      \u003ctd\u003e0.161716\u003c/td\u003e\n",
              "      \u003ctd\u003e0.078657\u003c/td\u003e\n",
              "      \u003ctd\u003e0.096484\u003c/td\u003e\n",
              "      \u003ctd\u003e0.045761\u003c/td\u003e\n",
              "      \u003ctd\u003e1537.137226\u003c/td\u003e\n",
              "      \u003ctd\u003e2172.213746\u003c/td\u003e\n",
              "      \u003ctd\u003e9\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e2172.213746\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003e[149, 32, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_6\u003c/td\u003e\n",
              "      \u003ctd\u003e4KK2_20150823_130650_314.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_11\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "      \u003ctd\u003e41883\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",
              "      \u003ctd\u003e6.830727\u003c/td\u003e\n",
              "      \u003ctd\u003e3.382573\u003c/td\u003e\n",
              "      \u003ctd\u003e1.990106\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.997969\u003c/td\u003e\n",
              "      \u003ctd\u003e0.145104\u003c/td\u003e\n",
              "      \u003ctd\u003e0.050339\u003c/td\u003e\n",
              "      \u003ctd\u003e0.072111\u003c/td\u003e\n",
              "      \u003ctd\u003e0.019148\u003c/td\u003e\n",
              "      \u003ctd\u003e1972.598657\u003c/td\u003e\n",
              "      \u003ctd\u003e2291.390798\u003c/td\u003e\n",
              "      \u003ctd\u003e13\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e2291.390798\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003e[1089, 489, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_11\u003c/td\u003e\n",
              "      \u003ctd\u003e6G7M_20150404_121844_208.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_17\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "      \u003ctd\u003e48385\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",
              "      \u003ctd\u003e1.760315\u003c/td\u003e\n",
              "      \u003ctd\u003e0.828016\u003c/td\u003e\n",
              "      \u003ctd\u003e1.656251\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.999264\u003c/td\u003e\n",
              "      \u003ctd\u003e0.030033\u003c/td\u003e\n",
              "      \u003ctd\u003e0.030637\u003c/td\u003e\n",
              "      \u003ctd\u003e0.010733\u003c/td\u003e\n",
              "      \u003ctd\u003e0.011180\u003c/td\u003e\n",
              "      \u003ctd\u003e2268.583209\u003c/td\u003e\n",
              "      \u003ctd\u003e2309.359159\u003c/td\u003e\n",
              "      \u003ctd\u003e4\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e2309.359159\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003e[433, 286, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_14\u003c/td\u003e\n",
              "      \u003ctd\u003emichael-niessl-KraoHdRYrRE-unsplash.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_11\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e36774\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1080]\u003c/td\u003e\n",
              "      \u003ctd\u003e4.110695\u003c/td\u003e\n",
              "      \u003ctd\u003e1.995982\u003c/td\u003e\n",
              "      \u003ctd\u003e2.432354\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998220\u003c/td\u003e\n",
              "      \u003ctd\u003e0.071748\u003c/td\u003e\n",
              "      \u003ctd\u003e0.037420\u003c/td\u003e\n",
              "      \u003ctd\u003e0.043709\u003c/td\u003e\n",
              "      \u003ctd\u003e0.019095\u003c/td\u003e\n",
              "      \u003ctd\u003e1991.945557\u003c/td\u003e\n",
              "      \u003ctd\u003e2241.368691\u003c/td\u003e\n",
              "      \u003ctd\u003e10\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e2241.368691\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003e[4, 99, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_6\u003c/td\u003e\n",
              "      \u003ctd\u003e0127_20161022_144117_906.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_14\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e22468\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1440]\u003c/td\u003e\n",
              "      \u003ctd\u003e5.866006\u003c/td\u003e\n",
              "      \u003ctd\u003e2.645947\u003c/td\u003e\n",
              "      \u003ctd\u003e3.550395\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.994369\u003c/td\u003e\n",
              "      \u003ctd\u003e0.095547\u003c/td\u003e\n",
              "      \u003ctd\u003e0.069588\u003c/td\u003e\n",
              "      \u003ctd\u003e0.049872\u003c/td\u003e\n",
              "      \u003ctd\u003e0.035815\u003c/td\u003e\n",
              "      \u003ctd\u003e1851.898409\u003c/td\u003e\n",
              "      \u003ctd\u003e1945.269660\u003c/td\u003e\n",
              "      \u003ctd\u003e9\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e1945.269660\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e...\u003c/th\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e12975\u003c/th\u003e\n",
              "      \u003ctd\u003e[361, 268, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_23\u003c/td\u003e\n",
              "      \u003ctd\u003e4KK2_20150912_125525_384.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_21\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "      \u003ctd\u003e15533\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",
              "      \u003ctd\u003e2.678031\u003c/td\u003e\n",
              "      \u003ctd\u003e1.201943\u003c/td\u003e\n",
              "      \u003ctd\u003e3.711323\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.988685\u003c/td\u003e\n",
              "      \u003ctd\u003e0.076957\u003c/td\u003e\n",
              "      \u003ctd\u003e0.128533\u003c/td\u003e\n",
              "      \u003ctd\u003e0.025589\u003c/td\u003e\n",
              "      \u003ctd\u003e0.051236\u003c/td\u003e\n",
              "      \u003ctd\u003e2334.257408\u003c/td\u003e\n",
              "      \u003ctd\u003e2105.427273\u003c/td\u003e\n",
              "      \u003ctd\u003e7\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e2334.257408\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e12976\u003c/th\u003e\n",
              "      \u003ctd\u003e[374, 425, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_23\u003c/td\u003e\n",
              "      \u003ctd\u003ejustin-lim-OiFRMM3o7l0-unsplash.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_22\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "      \u003ctd\u003e7306\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1440]\u003c/td\u003e\n",
              "      \u003ctd\u003e2.273484\u003c/td\u003e\n",
              "      \u003ctd\u003e1.044854\u003c/td\u003e\n",
              "      \u003ctd\u003e2.454072\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.995344\u003c/td\u003e\n",
              "      \u003ctd\u003e0.060886\u003c/td\u003e\n",
              "      \u003ctd\u003e0.078061\u003c/td\u003e\n",
              "      \u003ctd\u003e0.024588\u003c/td\u003e\n",
              "      \u003ctd\u003e0.033907\u003c/td\u003e\n",
              "      \u003ctd\u003e2347.775076\u003c/td\u003e\n",
              "      \u003ctd\u003e2209.434355\u003c/td\u003e\n",
              "      \u003ctd\u003e4\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e2347.775076\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e12977\u003c/th\u003e\n",
              "      \u003ctd\u003e[1093, 224, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_9\u003c/td\u003e\n",
              "      \u003ctd\u003e0127_20161022_144117_906.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_23\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e12069\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1440]\u003c/td\u003e\n",
              "      \u003ctd\u003e2.618022\u003c/td\u003e\n",
              "      \u003ctd\u003e1.303337\u003c/td\u003e\n",
              "      \u003ctd\u003e1.548064\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998452\u003c/td\u003e\n",
              "      \u003ctd\u003e0.050813\u003c/td\u003e\n",
              "      \u003ctd\u003e0.037381\u003c/td\u003e\n",
              "      \u003ctd\u003e0.023579\u003c/td\u003e\n",
              "      \u003ctd\u003e0.016908\u003c/td\u003e\n",
              "      \u003ctd\u003e2187.406607\u003c/td\u003e\n",
              "      \u003ctd\u003e2373.387899\u003c/td\u003e\n",
              "      \u003ctd\u003e12\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e2373.387899\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e12978\u003c/th\u003e\n",
              "      \u003ctd\u003e[887, 59, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_23\u003c/td\u003e\n",
              "      \u003ctd\u003e6G7M_20150404_121844_208.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_40\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "      \u003ctd\u003e7983\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",
              "      \u003ctd\u003e1.877585\u003c/td\u003e\n",
              "      \u003ctd\u003e0.923585\u003c/td\u003e\n",
              "      \u003ctd\u003e2.143382\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998512\u003c/td\u003e\n",
              "      \u003ctd\u003e0.051131\u003c/td\u003e\n",
              "      \u003ctd\u003e0.048955\u003c/td\u003e\n",
              "      \u003ctd\u003e0.020006\u003c/td\u003e\n",
              "      \u003ctd\u003e0.017322\u003c/td\u003e\n",
              "      \u003ctd\u003e2356.875932\u003c/td\u003e\n",
              "      \u003ctd\u003e2388.690716\u003c/td\u003e\n",
              "      \u003ctd\u003e6\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e2388.690716\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e12979\u003c/th\u003e\n",
              "      \u003ctd\u003e[1121, 175, 768, 768]\u003c/td\u003e\n",
              "      \u003ctd\u003edist_32\u003c/td\u003e\n",
              "      \u003ctd\u003e47L8_20150518_144020_604.png\u003c/td\u003e\n",
              "      \u003ctd\u003edist_30\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e1702517\u003c/td\u003e\n",
              "      \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",
              "      \u003ctd\u003e4.943964\u003c/td\u003e\n",
              "      \u003ctd\u003e2.460808\u003c/td\u003e\n",
              "      \u003ctd\u003e8.950997\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e0.955344\u003c/td\u003e\n",
              "      \u003ctd\u003e0.127071\u003c/td\u003e\n",
              "      \u003ctd\u003e0.201221\u003c/td\u003e\n",
              "      \u003ctd\u003e0.071460\u003c/td\u003e\n",
              "      \u003ctd\u003e0.155907\u003c/td\u003e\n",
              "      \u003ctd\u003e1937.790267\u003c/td\u003e\n",
              "      \u003ctd\u003e1402.672706\u003c/td\u003e\n",
              "      \u003ctd\u003e5\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "      \u003ctd\u003e1937.790267\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003cp\u003e12402 rows × 26 columns\u003c/p\u003e\n",
              "\u003c/div\u003e\n",
              "      \u003cbutton class=\"colab-df-convert\" onclick=\"convertToInteractive('df-73737e54-fd5b-44a9-a4dc-5e3c92c1719d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\"\u003e\n",
              "        \n",
              "  \u003csvg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\"\u003e\n",
              "    \u003cpath d=\"M0 0h24v24H0V0z\" fill=\"none\"/\u003e\n",
              "    \u003cpath d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/\u003e\u003cpath d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/\u003e\n",
              "  \u003c/svg\u003e\n",
              "      \u003c/button\u003e\n",
              "      \n",
              "  \u003cstyle\u003e\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  \u003c/style\u003e\n",
              "\n",
              "      \u003cscript\u003e\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-73737e54-fd5b-44a9-a4dc-5e3c92c1719d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-73737e54-fd5b-44a9-a4dc-5e3c92c1719d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '\u003ca target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb\u003edata table notebook\u003c/a\u003e'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      \u003c/script\u003e\n",
              "    \u003c/div\u003e\n",
              "  \u003c/div\u003e\n",
              "  "
            ],
            "text/plain": [
              "                        crop  greater  \\\n",
              "0       [176, 279, 768, 768]   dist_6   \n",
              "1        [149, 32, 768, 768]   dist_6   \n",
              "2      [1089, 489, 768, 768]  dist_11   \n",
              "3       [433, 286, 768, 768]  dist_14   \n",
              "4          [4, 99, 768, 768]   dist_6   \n",
              "...                      ...      ...   \n",
              "12975   [361, 268, 768, 768]  dist_23   \n",
              "12976   [374, 425, 768, 768]  dist_23   \n",
              "12977  [1093, 224, 768, 768]   dist_9   \n",
              "12978    [887, 59, 768, 768]  dist_23   \n",
              "12979  [1121, 175, 768, 768]  dist_32   \n",
              "\n",
              "                                          image   lesser  random_choice  \\\n",
              "0      edgar-castrejon-J_NbCcGPBOw-unsplash.png  dist_17          False   \n",
              "1                  4KK2_20150823_130650_314.png  dist_11          False   \n",
              "2                  6G7M_20150404_121844_208.png  dist_17          False   \n",
              "3       michael-niessl-KraoHdRYrRE-unsplash.png  dist_11           True   \n",
              "4                  0127_20161022_144117_906.png  dist_14           True   \n",
              "...                                         ...      ...            ...   \n",
              "12975              4KK2_20150912_125525_384.png  dist_21          False   \n",
              "12976       justin-lim-OiFRMM3o7l0-unsplash.png  dist_22          False   \n",
              "12977              0127_20161022_144117_906.png  dist_23           True   \n",
              "12978              6G7M_20150404_121844_208.png  dist_40          False   \n",
              "12979              47L8_20150518_144020_604.png  dist_30           True   \n",
              "\n",
              "       rater_time_ms    image_dims  greater_butteraugli_max  \\\n",
              "0              75125  [1280, 1920]                 6.792748   \n",
              "1              41883  [1920, 1424]                 6.830727   \n",
              "2              48385  [1920, 1424]                 1.760315   \n",
              "3              36774  [1920, 1080]                 4.110695   \n",
              "4              22468  [1920, 1440]                 5.866006   \n",
              "...              ...           ...                      ...   \n",
              "12975          15533  [1920, 1424]                 2.678031   \n",
              "12976           7306  [1920, 1440]                 2.273484   \n",
              "12977          12069  [1920, 1440]                 2.618022   \n",
              "12978           7983  [1920, 1424]                 1.877585   \n",
              "12979        1702517  [1920, 1424]                 4.943964   \n",
              "\n",
              "       greater_butteraugli_6  lesser_butteraugli_max  ...  lesser_msssim-y  \\\n",
              "0                   3.291877                2.287096  ...         0.993973   \n",
              "1                   3.382573                1.990106  ...         0.997969   \n",
              "2                   0.828016                1.656251  ...         0.999264   \n",
              "3                   1.995982                2.432354  ...         0.998220   \n",
              "4                   2.645947                3.550395  ...         0.994369   \n",
              "...                      ...                     ...  ...              ...   \n",
              "12975               1.201943                3.711323  ...         0.988685   \n",
              "12976               1.044854                2.454072  ...         0.995344   \n",
              "12977               1.303337                1.548064  ...         0.998452   \n",
              "12978               0.923585                2.143382  ...         0.998512   \n",
              "12979               2.460808                8.950997  ...         0.955344   \n",
              "\n",
              "       greater_nlpd-y  lesser_nlpd-y  greater_ssimulacra  lesser_ssimulacra  \\\n",
              "0            0.161716       0.078657            0.096484           0.045761   \n",
              "1            0.145104       0.050339            0.072111           0.019148   \n",
              "2            0.030033       0.030637            0.010733           0.011180   \n",
              "3            0.071748       0.037420            0.043709           0.019095   \n",
              "4            0.095547       0.069588            0.049872           0.035815   \n",
              "...               ...            ...                 ...                ...   \n",
              "12975        0.076957       0.128533            0.025589           0.051236   \n",
              "12976        0.060886       0.078061            0.024588           0.033907   \n",
              "12977        0.050813       0.037381            0.023579           0.016908   \n",
              "12978        0.051131       0.048955            0.020006           0.017322   \n",
              "12979        0.127071       0.201221            0.071460           0.155907   \n",
              "\n",
              "       greater_elo   lesser_elo  rater_flips  complete_crop    worst_elo  \n",
              "0      1537.137226  2172.213746            9           True  2172.213746  \n",
              "1      1972.598657  2291.390798           13           True  2291.390798  \n",
              "2      2268.583209  2309.359159            4           True  2309.359159  \n",
              "3      1991.945557  2241.368691           10           True  2241.368691  \n",
              "4      1851.898409  1945.269660            9           True  1945.269660  \n",
              "...            ...          ...          ...            ...          ...  \n",
              "12975  2334.257408  2105.427273            7           True  2334.257408  \n",
              "12976  2347.775076  2209.434355            4           True  2347.775076  \n",
              "12977  2187.406607  2373.387899           12           True  2373.387899  \n",
              "12978  2356.875932  2388.690716            6           True  2388.690716  \n",
              "12979  1937.790267  1402.672706            5           True  1937.790267  \n",
              "\n",
              "[12402 rows x 26 columns]"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "data['complete_crop'] = data.apply(lambda row: row.crop[0] + row.crop[2] \u003c= row.image_dims[0] and row.crop[1] + row.crop[3] \u003c= row.image_dims[1], axis=1)\n",
        "data['worst_elo'] = data.apply(lambda row: row.greater_elo if row.greater_elo \u003e row.lesser_elo else row.lesser_elo, axis=1)\n",
        "data = data[(data.rater_flips \u003e 2) \u0026 (data.rater_time_ms \u003e 3000) \u0026 (data.complete_crop == True)]\n",
        "data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RNT0sDkLQJnd"
      },
      "outputs": [],
      "source": [
        "def strip(ary, n):\n",
        "  def stripfun(sum, el):\n",
        "    sum[el] = el[n:]\n",
        "    return sum\n",
        "  return functools.reduce(stripfun, ary, {})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "63zgtekaR4fj"
      },
      "outputs": [],
      "source": [
        "greater_metric_cols = list(filter(lambda el: el.startswith('greater_') and not el.endswith('_file'), list(data.columns)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Kt71n9RsS1E-"
      },
      "outputs": [],
      "source": [
        "lesser_metric_cols = list(filter(lambda el: el.startswith('lesser_') and not el.endswith('_file'), list(data.columns)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dRTUD1m6PTlq"
      },
      "outputs": [],
      "source": [
        "greater_metrics = data[greater_metric_cols]\n",
        "greater_metrics = greater_metrics.rename(columns=strip(greater_metric_cols, 8))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iDOc7lrzS-2n"
      },
      "outputs": [],
      "source": [
        "lesser_metrics = data[lesser_metric_cols]\n",
        "lesser_metrics = lesser_metrics.rename(columns=strip(lesser_metric_cols, 7))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ZyN9BMbzZQb"
      },
      "source": [
        "To allow a rank correlation, like Spearman, combine the metrics of the worse distortion (lesser), and the better distortion (greater), into one dataframe. To also allow comparing correlation in different regions of quality, sort by ELO score."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "id": "8flMfeykR1wL",
        "outputId": "39d9cb1b-2bd1-445b-bf83-ad61b15179ab"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  \u003cdiv id=\"df-ede9f7e9-1552-4956-af74-848c6cd1bfaa\"\u003e\n",
              "    \u003cdiv class=\"colab-df-container\"\u003e\n",
              "      \u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003ebutteraugli_max\u003c/th\u003e\n",
              "      \u003cth\u003ebutteraugli_6\u003c/th\u003e\n",
              "      \u003cth\u003efsim-y\u003c/th\u003e\n",
              "      \u003cth\u003efsim-rgb\u003c/th\u003e\n",
              "      \u003cth\u003emsssim-y\u003c/th\u003e\n",
              "      \u003cth\u003enlpd-y\u003c/th\u003e\n",
              "      \u003cth\u003essimulacra\u003c/th\u003e\n",
              "      \u003cth\u003eelo\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003e8.867950\u003c/td\u003e\n",
              "      \u003ctd\u003e4.764297\u003c/td\u003e\n",
              "      \u003ctd\u003e0.978342\u003c/td\u003e\n",
              "      \u003ctd\u003e0.974386\u003c/td\u003e\n",
              "      \u003ctd\u003e0.965125\u003c/td\u003e\n",
              "      \u003ctd\u003e0.242242\u003c/td\u003e\n",
              "      \u003ctd\u003e0.144204\u003c/td\u003e\n",
              "      \u003ctd\u003e824.179269\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003e8.879775\u003c/td\u003e\n",
              "      \u003ctd\u003e4.580521\u003c/td\u003e\n",
              "      \u003ctd\u003e0.979431\u003c/td\u003e\n",
              "      \u003ctd\u003e0.976825\u003c/td\u003e\n",
              "      \u003ctd\u003e0.960813\u003c/td\u003e\n",
              "      \u003ctd\u003e0.228356\u003c/td\u003e\n",
              "      \u003ctd\u003e0.125745\u003c/td\u003e\n",
              "      \u003ctd\u003e824.179269\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003e8.815258\u003c/td\u003e\n",
              "      \u003ctd\u003e4.479465\u003c/td\u003e\n",
              "      \u003ctd\u003e0.980187\u003c/td\u003e\n",
              "      \u003ctd\u003e0.977970\u003c/td\u003e\n",
              "      \u003ctd\u003e0.961923\u003c/td\u003e\n",
              "      \u003ctd\u003e0.215990\u003c/td\u003e\n",
              "      \u003ctd\u003e0.119023\u003c/td\u003e\n",
              "      \u003ctd\u003e824.179269\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003e8.913489\u003c/td\u003e\n",
              "      \u003ctd\u003e4.727989\u003c/td\u003e\n",
              "      \u003ctd\u003e0.976972\u003c/td\u003e\n",
              "      \u003ctd\u003e0.973410\u003c/td\u003e\n",
              "      \u003ctd\u003e0.959204\u003c/td\u003e\n",
              "      \u003ctd\u003e0.240950\u003c/td\u003e\n",
              "      \u003ctd\u003e0.142649\u003c/td\u003e\n",
              "      \u003ctd\u003e824.179269\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003e9.732282\u003c/td\u003e\n",
              "      \u003ctd\u003e4.798093\u003c/td\u003e\n",
              "      \u003ctd\u003e0.977469\u003c/td\u003e\n",
              "      \u003ctd\u003e0.973566\u003c/td\u003e\n",
              "      \u003ctd\u003e0.958279\u003c/td\u003e\n",
              "      \u003ctd\u003e0.250352\u003c/td\u003e\n",
              "      \u003ctd\u003e0.148506\u003c/td\u003e\n",
              "      \u003ctd\u003e824.179269\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e...\u003c/th\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "      \u003ctd\u003e...\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e24799\u003c/th\u003e\n",
              "      \u003ctd\u003e1.762231\u003c/td\u003e\n",
              "      \u003ctd\u003e0.863336\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998921\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998753\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998144\u003c/td\u003e\n",
              "      \u003ctd\u003e0.053619\u003c/td\u003e\n",
              "      \u003ctd\u003e0.014470\u003c/td\u003e\n",
              "      \u003ctd\u003e2699.407630\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e24800\u003c/th\u003e\n",
              "      \u003ctd\u003e1.857362\u003c/td\u003e\n",
              "      \u003ctd\u003e0.921340\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998864\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998748\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998029\u003c/td\u003e\n",
              "      \u003ctd\u003e0.051740\u003c/td\u003e\n",
              "      \u003ctd\u003e0.016668\u003c/td\u003e\n",
              "      \u003ctd\u003e2699.407630\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e24801\u003c/th\u003e\n",
              "      \u003ctd\u003e2.087677\u003c/td\u003e\n",
              "      \u003ctd\u003e0.921140\u003c/td\u003e\n",
              "      \u003ctd\u003e0.999216\u003c/td\u003e\n",
              "      \u003ctd\u003e0.999160\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998273\u003c/td\u003e\n",
              "      \u003ctd\u003e0.042458\u003c/td\u003e\n",
              "      \u003ctd\u003e0.013809\u003c/td\u003e\n",
              "      \u003ctd\u003e2699.407630\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e24802\u003c/th\u003e\n",
              "      \u003ctd\u003e1.668185\u003c/td\u003e\n",
              "      \u003ctd\u003e0.847164\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998240\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998118\u003c/td\u003e\n",
              "      \u003ctd\u003e0.996968\u003c/td\u003e\n",
              "      \u003ctd\u003e0.056177\u003c/td\u003e\n",
              "      \u003ctd\u003e0.016632\u003c/td\u003e\n",
              "      \u003ctd\u003e2699.407630\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e24803\u003c/th\u003e\n",
              "      \u003ctd\u003e1.859033\u003c/td\u003e\n",
              "      \u003ctd\u003e0.889231\u003c/td\u003e\n",
              "      \u003ctd\u003e0.999222\u003c/td\u003e\n",
              "      \u003ctd\u003e0.999119\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998511\u003c/td\u003e\n",
              "      \u003ctd\u003e0.045151\u003c/td\u003e\n",
              "      \u003ctd\u003e0.015015\u003c/td\u003e\n",
              "      \u003ctd\u003e2699.407630\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003cp\u003e24804 rows × 8 columns\u003c/p\u003e\n",
              "\u003c/div\u003e\n",
              "      \u003cbutton class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ede9f7e9-1552-4956-af74-848c6cd1bfaa')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\"\u003e\n",
              "        \n",
              "  \u003csvg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\"\u003e\n",
              "    \u003cpath d=\"M0 0h24v24H0V0z\" fill=\"none\"/\u003e\n",
              "    \u003cpath d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/\u003e\u003cpath d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/\u003e\n",
              "  \u003c/svg\u003e\n",
              "      \u003c/button\u003e\n",
              "      \n",
              "  \u003cstyle\u003e\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  \u003c/style\u003e\n",
              "\n",
              "      \u003cscript\u003e\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ede9f7e9-1552-4956-af74-848c6cd1bfaa button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ede9f7e9-1552-4956-af74-848c6cd1bfaa');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '\u003ca target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb\u003edata table notebook\u003c/a\u003e'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      \u003c/script\u003e\n",
              "    \u003c/div\u003e\n",
              "  \u003c/div\u003e\n",
              "  "
            ],
            "text/plain": [
              "       butteraugli_max  butteraugli_6    fsim-y  fsim-rgb  msssim-y    nlpd-y  \\\n",
              "0             8.867950       4.764297  0.978342  0.974386  0.965125  0.242242   \n",
              "1             8.879775       4.580521  0.979431  0.976825  0.960813  0.228356   \n",
              "2             8.815258       4.479465  0.980187  0.977970  0.961923  0.215990   \n",
              "3             8.913489       4.727989  0.976972  0.973410  0.959204  0.240950   \n",
              "4             9.732282       4.798093  0.977469  0.973566  0.958279  0.250352   \n",
              "...                ...            ...       ...       ...       ...       ...   \n",
              "24799         1.762231       0.863336  0.998921  0.998753  0.998144  0.053619   \n",
              "24800         1.857362       0.921340  0.998864  0.998748  0.998029  0.051740   \n",
              "24801         2.087677       0.921140  0.999216  0.999160  0.998273  0.042458   \n",
              "24802         1.668185       0.847164  0.998240  0.998118  0.996968  0.056177   \n",
              "24803         1.859033       0.889231  0.999222  0.999119  0.998511  0.045151   \n",
              "\n",
              "       ssimulacra          elo  \n",
              "0        0.144204   824.179269  \n",
              "1        0.125745   824.179269  \n",
              "2        0.119023   824.179269  \n",
              "3        0.142649   824.179269  \n",
              "4        0.148506   824.179269  \n",
              "...           ...          ...  \n",
              "24799    0.014470  2699.407630  \n",
              "24800    0.016668  2699.407630  \n",
              "24801    0.013809  2699.407630  \n",
              "24802    0.016632  2699.407630  \n",
              "24803    0.015015  2699.407630  \n",
              "\n",
              "[24804 rows x 8 columns]"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "metrics = pd.concat([greater_metrics, lesser_metrics])\n",
        "metrics = metrics.sort_values('elo').reset_index(drop=True)\n",
        "metrics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gKClTtjRzjj_"
      },
      "source": [
        "Then compute the correlation matrix for these, using Spearman's rank correlation coeffient."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "id": "8bTKAKn6ToJD",
        "outputId": "b355b0a9-1f06-435a-da1f-3ec3c1cbe552"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  \u003cdiv id=\"df-df0a5791-d639-462a-94ac-26e812c3d0d1\"\u003e\n",
              "    \u003cdiv class=\"colab-df-container\"\u003e\n",
              "      \u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003ebutteraugli_max\u003c/th\u003e\n",
              "      \u003cth\u003ebutteraugli_6\u003c/th\u003e\n",
              "      \u003cth\u003efsim-y\u003c/th\u003e\n",
              "      \u003cth\u003efsim-rgb\u003c/th\u003e\n",
              "      \u003cth\u003emsssim-y\u003c/th\u003e\n",
              "      \u003cth\u003enlpd-y\u003c/th\u003e\n",
              "      \u003cth\u003essimulacra\u003c/th\u003e\n",
              "      \u003cth\u003eelo\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003ebutteraugli_max\u003c/th\u003e\n",
              "      \u003ctd\u003e1.000000\u003c/td\u003e\n",
              "      \u003ctd\u003e0.986256\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.886318\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.892921\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.823469\u003c/td\u003e\n",
              "      \u003ctd\u003e0.792767\u003c/td\u003e\n",
              "      \u003ctd\u003e0.842150\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.874967\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003ebutteraugli_6\u003c/th\u003e\n",
              "      \u003ctd\u003e0.986256\u003c/td\u003e\n",
              "      \u003ctd\u003e1.000000\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.877144\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.883633\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.833595\u003c/td\u003e\n",
              "      \u003ctd\u003e0.807155\u003c/td\u003e\n",
              "      \u003ctd\u003e0.859254\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.863846\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003efsim-y\u003c/th\u003e\n",
              "      \u003ctd\u003e-0.886318\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.877144\u003c/td\u003e\n",
              "      \u003ctd\u003e1.000000\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998044\u003c/td\u003e\n",
              "      \u003ctd\u003e0.900061\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.845079\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.840729\u003c/td\u003e\n",
              "      \u003ctd\u003e0.859066\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003efsim-rgb\u003c/th\u003e\n",
              "      \u003ctd\u003e-0.892921\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.883633\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998044\u003c/td\u003e\n",
              "      \u003ctd\u003e1.000000\u003c/td\u003e\n",
              "      \u003ctd\u003e0.898808\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.850361\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.848226\u003c/td\u003e\n",
              "      \u003ctd\u003e0.862459\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003emsssim-y\u003c/th\u003e\n",
              "      \u003ctd\u003e-0.823469\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.833595\u003c/td\u003e\n",
              "      \u003ctd\u003e0.900061\u003c/td\u003e\n",
              "      \u003ctd\u003e0.898808\u003c/td\u003e\n",
              "      \u003ctd\u003e1.000000\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.958732\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.942831\u003c/td\u003e\n",
              "      \u003ctd\u003e0.766435\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003enlpd-y\u003c/th\u003e\n",
              "      \u003ctd\u003e0.792767\u003c/td\u003e\n",
              "      \u003ctd\u003e0.807155\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.845079\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.850361\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.958732\u003c/td\u003e\n",
              "      \u003ctd\u003e1.000000\u003c/td\u003e\n",
              "      \u003ctd\u003e0.933383\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.699681\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003essimulacra\u003c/th\u003e\n",
              "      \u003ctd\u003e0.842150\u003c/td\u003e\n",
              "      \u003ctd\u003e0.859254\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.840729\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.848226\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.942831\u003c/td\u003e\n",
              "      \u003ctd\u003e0.933383\u003c/td\u003e\n",
              "      \u003ctd\u003e1.000000\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.766894\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003eelo\u003c/th\u003e\n",
              "      \u003ctd\u003e-0.874967\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.863846\u003c/td\u003e\n",
              "      \u003ctd\u003e0.859066\u003c/td\u003e\n",
              "      \u003ctd\u003e0.862459\u003c/td\u003e\n",
              "      \u003ctd\u003e0.766435\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.699681\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.766894\u003c/td\u003e\n",
              "      \u003ctd\u003e1.000000\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e\n",
              "      \u003cbutton class=\"colab-df-convert\" onclick=\"convertToInteractive('df-df0a5791-d639-462a-94ac-26e812c3d0d1')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\"\u003e\n",
              "        \n",
              "  \u003csvg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\"\u003e\n",
              "    \u003cpath d=\"M0 0h24v24H0V0z\" fill=\"none\"/\u003e\n",
              "    \u003cpath d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/\u003e\u003cpath d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/\u003e\n",
              "  \u003c/svg\u003e\n",
              "      \u003c/button\u003e\n",
              "      \n",
              "  \u003cstyle\u003e\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  \u003c/style\u003e\n",
              "\n",
              "      \u003cscript\u003e\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-df0a5791-d639-462a-94ac-26e812c3d0d1 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-df0a5791-d639-462a-94ac-26e812c3d0d1');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '\u003ca target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb\u003edata table notebook\u003c/a\u003e'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      \u003c/script\u003e\n",
              "    \u003c/div\u003e\n",
              "  \u003c/div\u003e\n",
              "  "
            ],
            "text/plain": [
              "                 butteraugli_max  butteraugli_6    fsim-y  fsim-rgb  msssim-y  \\\n",
              "butteraugli_max         1.000000       0.986256 -0.886318 -0.892921 -0.823469   \n",
              "butteraugli_6           0.986256       1.000000 -0.877144 -0.883633 -0.833595   \n",
              "fsim-y                 -0.886318      -0.877144  1.000000  0.998044  0.900061   \n",
              "fsim-rgb               -0.892921      -0.883633  0.998044  1.000000  0.898808   \n",
              "msssim-y               -0.823469      -0.833595  0.900061  0.898808  1.000000   \n",
              "nlpd-y                  0.792767       0.807155 -0.845079 -0.850361 -0.958732   \n",
              "ssimulacra              0.842150       0.859254 -0.840729 -0.848226 -0.942831   \n",
              "elo                    -0.874967      -0.863846  0.859066  0.862459  0.766435   \n",
              "\n",
              "                   nlpd-y  ssimulacra       elo  \n",
              "butteraugli_max  0.792767    0.842150 -0.874967  \n",
              "butteraugli_6    0.807155    0.859254 -0.863846  \n",
              "fsim-y          -0.845079   -0.840729  0.859066  \n",
              "fsim-rgb        -0.850361   -0.848226  0.862459  \n",
              "msssim-y        -0.958732   -0.942831  0.766435  \n",
              "nlpd-y           1.000000    0.933383 -0.699681  \n",
              "ssimulacra       0.933383    1.000000 -0.766894  \n",
              "elo             -0.699681   -0.766894  1.000000  "
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "corrs = metrics.corr(method='spearman')\n",
        "corrs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tk-8SPyiD7Bd"
      },
      "outputs": [],
      "source": [
        "metric_cols = list(map(lambda name: name[7:], lesser_metric_cols))\n",
        "metric_cols.remove('elo')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VAdFdjD0Dtmb"
      },
      "outputs": [],
      "source": [
        "def rollingcorr(df, method, window_size, step_size):\n",
        "  res = []\n",
        "  for start in range(0, df.shape[0] - window_size, step_size):\n",
        "    window = df[start:start+window_size]\n",
        "    row = [window.iloc[-1]['elo']]\n",
        "    for metric_name in metric_cols:\n",
        "      row.append(np.abs(window[metric_name].corr(window['elo'], method=method)))\n",
        "    res.append(row)\n",
        "  return pd.DataFrame(res, dtype=np.float, columns=['elo'] + list(map(lambda name: f\"{name}\", metric_cols)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p5axWasVGP1E"
      },
      "source": [
        "Plot the correlation in a rolling window of 5000 evaluations with a step of 1000 evaluations for each metric, to see how they behave across a range of ELO scores."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 506
        },
        "id": "kBEIDlJDEIIo",
        "outputId": "c7ed5ac9-30f3-4da3-896e-912380e5955b"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:9: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
            "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
            "  if __name__ == '__main__':\n"
          ]
        },
        {
          "data": {
            "application/javascript": [
              "\n",
              "(function(root) {\n",
              "  function now() {\n",
              "    return new Date();\n",
              "  }\n",
              "\n",
              "  var force = true;\n",
              "\n",
              "  if (typeof root._bokeh_onload_callbacks === \"undefined\" || force === true) {\n",
              "    root._bokeh_onload_callbacks = [];\n",
              "    root._bokeh_is_loading = undefined;\n",
              "  }\n",
              "\n",
              "  var JS_MIME_TYPE = 'application/javascript';\n",
              "  var HTML_MIME_TYPE = 'text/html';\n",
              "  var EXEC_MIME_TYPE = 'application/vnd.bokehjs_exec.v0+json';\n",
              "  var CLASS_NAME = 'output_bokeh rendered_html';\n",
              "\n",
              "  /**\n",
              "   * Render data to the DOM node\n",
              "   */\n",
              "  function render(props, node) {\n",
              "    var script = document.createElement(\"script\");\n",
              "    node.appendChild(script);\n",
              "  }\n",
              "\n",
              "  /**\n",
              "   * Handle when an output is cleared or removed\n",
              "   */\n",
              "  function handleClearOutput(event, handle) {\n",
              "    var cell = handle.cell;\n",
              "\n",
              "    var id = cell.output_area._bokeh_element_id;\n",
              "    var server_id = cell.output_area._bokeh_server_id;\n",
              "    // Clean up Bokeh references\n",
              "    if (id != null \u0026\u0026 id in Bokeh.index) {\n",
              "      Bokeh.index[id].model.document.clear();\n",
              "      delete Bokeh.index[id];\n",
              "    }\n",
              "\n",
              "    if (server_id !== undefined) {\n",
              "      // Clean up Bokeh references\n",
              "      var cmd = \"from bokeh.io.state import curstate; print(curstate().uuid_to_server['\" + server_id + \"'].get_sessions()[0].document.roots[0]._id)\";\n",
              "      cell.notebook.kernel.execute(cmd, {\n",
              "        iopub: {\n",
              "          output: function(msg) {\n",
              "            var id = msg.content.text.trim();\n",
              "            if (id in Bokeh.index) {\n",
              "              Bokeh.index[id].model.document.clear();\n",
              "              delete Bokeh.index[id];\n",
              "            }\n",
              "          }\n",
              "        }\n",
              "      });\n",
              "      // Destroy server and session\n",
              "      var cmd = \"import bokeh.io.notebook as ion; ion.destroy_server('\" + server_id + \"')\";\n",
              "      cell.notebook.kernel.execute(cmd);\n",
              "    }\n",
              "  }\n",
              "\n",
              "  /**\n",
              "   * Handle when a new output is added\n",
              "   */\n",
              "  function handleAddOutput(event, handle) {\n",
              "    var output_area = handle.output_area;\n",
              "    var output = handle.output;\n",
              "\n",
              "    // limit handleAddOutput to display_data with EXEC_MIME_TYPE content only\n",
              "    if ((output.output_type != \"display_data\") || (!Object.prototype.hasOwnProperty.call(output.data, EXEC_MIME_TYPE))) {\n",
              "      return\n",
              "    }\n",
              "\n",
              "    var toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n",
              "\n",
              "    if (output.metadata[EXEC_MIME_TYPE][\"id\"] !== undefined) {\n",
              "      toinsert[toinsert.length - 1].firstChild.textContent = output.data[JS_MIME_TYPE];\n",
              "      // store reference to embed id on output_area\n",
              "      output_area._bokeh_element_id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n",
              "    }\n",
              "    if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n",
              "      var bk_div = document.createElement(\"div\");\n",
              "      bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n",
              "      var script_attrs = bk_div.children[0].attributes;\n",
              "      for (var i = 0; i \u003c script_attrs.length; i++) {\n",
              "        toinsert[toinsert.length - 1].firstChild.setAttribute(script_attrs[i].name, script_attrs[i].value);\n",
              "        toinsert[toinsert.length - 1].firstChild.textContent = bk_div.children[0].textContent\n",
              "      }\n",
              "      // store reference to server id on output_area\n",
              "      output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n",
              "    }\n",
              "  }\n",
              "\n",
              "  function register_renderer(events, OutputArea) {\n",
              "\n",
              "    function append_mime(data, metadata, element) {\n",
              "      // create a DOM node to render to\n",
              "      var toinsert = this.create_output_subarea(\n",
              "        metadata,\n",
              "        CLASS_NAME,\n",
              "        EXEC_MIME_TYPE\n",
              "      );\n",
              "      this.keyboard_manager.register_events(toinsert);\n",
              "      // Render to node\n",
              "      var props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n",
              "      render(props, toinsert[toinsert.length - 1]);\n",
              "      element.append(toinsert);\n",
              "      return toinsert\n",
              "    }\n",
              "\n",
              "    /* Handle when an output is cleared or removed */\n",
              "    events.on('clear_output.CodeCell', handleClearOutput);\n",
              "    events.on('delete.Cell', handleClearOutput);\n",
              "\n",
              "    /* Handle when a new output is added */\n",
              "    events.on('output_added.OutputArea', handleAddOutput);\n",
              "\n",
              "    /**\n",
              "     * Register the mime type and append_mime function with output_area\n",
              "     */\n",
              "    OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n",
              "      /* Is output safe? */\n",
              "      safe: true,\n",
              "      /* Index of renderer in `output_area.display_order` */\n",
              "      index: 0\n",
              "    });\n",
              "  }\n",
              "\n",
              "  // register the mime type if in Jupyter Notebook environment and previously unregistered\n",
              "  if (root.Jupyter !== undefined) {\n",
              "    var events = require('base/js/events');\n",
              "    var OutputArea = require('notebook/js/outputarea').OutputArea;\n",
              "\n",
              "    if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n",
              "      register_renderer(events, OutputArea);\n",
              "    }\n",
              "  }\n",
              "\n",
              "  \n",
              "  if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n",
              "    root._bokeh_timeout = Date.now() + 5000;\n",
              "    root._bokeh_failed_load = false;\n",
              "  }\n",
              "\n",
              "  var NB_LOAD_WARNING = {'data': {'text/html':\n",
              "     \"\u003cdiv style='background-color: #fdd'\u003e\\n\"+\n",
              "     \"\u003cp\u003e\\n\"+\n",
              "     \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
              "     \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
              "     \"\u003c/p\u003e\\n\"+\n",
              "     \"\u003cul\u003e\\n\"+\n",
              "     \"\u003cli\u003ere-rerun `output_notebook()` to attempt to load from CDN again, or\u003c/li\u003e\\n\"+\n",
              "     \"\u003cli\u003euse INLINE resources instead, as so:\u003c/li\u003e\\n\"+\n",
              "     \"\u003c/ul\u003e\\n\"+\n",
              "     \"\u003ccode\u003e\\n\"+\n",
              "     \"from bokeh.resources import INLINE\\n\"+\n",
              "     \"output_notebook(resources=INLINE)\\n\"+\n",
              "     \"\u003c/code\u003e\\n\"+\n",
              "     \"\u003c/div\u003e\"}};\n",
              "\n",
              "  function display_loaded() {\n",
              "    var el = document.getElementById(null);\n",
              "    if (el != null) {\n",
              "      el.textContent = \"BokehJS is loading...\";\n",
              "    }\n",
              "    if (root.Bokeh !== undefined) {\n",
              "      if (el != null) {\n",
              "        el.textContent = \"BokehJS \" + root.Bokeh.version + \" successfully loaded.\";\n",
              "      }\n",
              "    } else if (Date.now() \u003c root._bokeh_timeout) {\n",
              "      setTimeout(display_loaded, 100)\n",
              "    }\n",
              "  }\n",
              "\n",
              "\n",
              "  function run_callbacks() {\n",
              "    try {\n",
              "      root._bokeh_onload_callbacks.forEach(function(callback) {\n",
              "        if (callback != null)\n",
              "          callback();\n",
              "      });\n",
              "    } finally {\n",
              "      delete root._bokeh_onload_callbacks\n",
              "    }\n",
              "    console.debug(\"Bokeh: all callbacks have finished\");\n",
              "  }\n",
              "\n",
              "  function load_libs(css_urls, js_urls, callback) {\n",
              "    if (css_urls == null) css_urls = [];\n",
              "    if (js_urls == null) js_urls = [];\n",
              "\n",
              "    root._bokeh_onload_callbacks.push(callback);\n",
              "    if (root._bokeh_is_loading \u003e 0) {\n",
              "      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
              "      return null;\n",
              "    }\n",
              "    if (js_urls == null || js_urls.length === 0) {\n",
              "      run_callbacks();\n",
              "      return null;\n",
              "    }\n",
              "    console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
              "    root._bokeh_is_loading = css_urls.length + js_urls.length;\n",
              "\n",
              "    function on_load() {\n",
              "      root._bokeh_is_loading--;\n",
              "      if (root._bokeh_is_loading === 0) {\n",
              "        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n",
              "        run_callbacks()\n",
              "      }\n",
              "    }\n",
              "\n",
              "    function on_error(url) {\n",
              "      console.error(\"failed to load \" + url);\n",
              "    }\n",
              "\n",
              "    for (let i = 0; i \u003c css_urls.length; i++) {\n",
              "      const url = css_urls[i];\n",
              "      const element = document.createElement(\"link\");\n",
              "      element.onload = on_load;\n",
              "      element.onerror = on_error.bind(null, url);\n",
              "      element.rel = \"stylesheet\";\n",
              "      element.type = \"text/css\";\n",
              "      element.href = url;\n",
              "      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n",
              "      document.body.appendChild(element);\n",
              "    }\n",
              "\n",
              "    const hashes = {\"https://cdn.bokeh.org/bokeh/release/bokeh-2.3.3.min.js\": \"dM3QQsP+wXdHg42wTqW85BjZQdLNNIXqlPw/BgKoExPmTG7ZLML4EGqLMfqHT6ON\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.3.3.min.js\": \"8x57I4YuIfu8XyZfFo0XVr2WAT8EK4rh/uDe3wF7YuW2FNUSNEpJbsPaB1nJ2fz2\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.3.3.min.js\": \"3QTqdz9LyAm2i0sG5XTePsHec3UHWwVsrOL68SYRoAXsafvfAyqtQ+h440+qIBhS\"};\n",
              "\n",
              "    for (let i = 0; i \u003c js_urls.length; i++) {\n",
              "      const url = js_urls[i];\n",
              "      const element = document.createElement('script');\n",
              "      element.onload = on_load;\n",
              "      element.onerror = on_error.bind(null, url);\n",
              "      element.async = false;\n",
              "      element.src = url;\n",
              "      if (url in hashes) {\n",
              "        element.crossOrigin = \"anonymous\";\n",
              "        element.integrity = \"sha384-\" + hashes[url];\n",
              "      }\n",
              "      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
              "      document.head.appendChild(element);\n",
              "    }\n",
              "  };\n",
              "\n",
              "  function inject_raw_css(css) {\n",
              "    const element = document.createElement(\"style\");\n",
              "    element.appendChild(document.createTextNode(css));\n",
              "    document.body.appendChild(element);\n",
              "  }\n",
              "\n",
              "  \n",
              "  var js_urls = [\"https://cdn.bokeh.org/bokeh/release/bokeh-2.3.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.3.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.3.3.min.js\"];\n",
              "  var css_urls = [];\n",
              "  \n",
              "\n",
              "  var inline_js = [\n",
              "    function(Bokeh) {\n",
              "      Bokeh.set_log_level(\"info\");\n",
              "    },\n",
              "    function(Bokeh) {\n",
              "    \n",
              "    \n",
              "    }\n",
              "  ];\n",
              "\n",
              "  function run_inline_js() {\n",
              "    \n",
              "    if (root.Bokeh !== undefined || force === true) {\n",
              "      \n",
              "    for (var i = 0; i \u003c inline_js.length; i++) {\n",
              "      inline_js[i].call(root, root.Bokeh);\n",
              "    }\n",
              "    } else if (Date.now() \u003c root._bokeh_timeout) {\n",
              "      setTimeout(run_inline_js, 100);\n",
              "    } else if (!root._bokeh_failed_load) {\n",
              "      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
              "      root._bokeh_failed_load = true;\n",
              "    } else if (force !== true) {\n",
              "      var cell = $(document.getElementById(null)).parents('.cell').data().cell;\n",
              "      cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",
              "    }\n",
              "\n",
              "  }\n",
              "\n",
              "  if (root._bokeh_is_loading === 0) {\n",
              "    console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
              "    run_inline_js();\n",
              "  } else {\n",
              "    load_libs(css_urls, js_urls, function() {\n",
              "      console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n",
              "      run_inline_js();\n",
              "    });\n",
              "  }\n",
              "}(window));"
            ],
            "application/vnd.bokehjs_load.v0+json": "\n(function(root) {\n  function now() {\n    return new Date();\n  }\n\n  var force = true;\n\n  if (typeof root._bokeh_onload_callbacks === \"undefined\" || force === true) {\n    root._bokeh_onload_callbacks = [];\n    root._bokeh_is_loading = undefined;\n  }\n\n  \n\n  \n  if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n    root._bokeh_timeout = Date.now() + 5000;\n    root._bokeh_failed_load = false;\n  }\n\n  var NB_LOAD_WARNING = {'data': {'text/html':\n     \"\u003cdiv style='background-color: #fdd'\u003e\\n\"+\n     \"\u003cp\u003e\\n\"+\n     \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n     \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n     \"\u003c/p\u003e\\n\"+\n     \"\u003cul\u003e\\n\"+\n     \"\u003cli\u003ere-rerun `output_notebook()` to attempt to load from CDN again, or\u003c/li\u003e\\n\"+\n     \"\u003cli\u003euse INLINE resources instead, as so:\u003c/li\u003e\\n\"+\n     \"\u003c/ul\u003e\\n\"+\n     \"\u003ccode\u003e\\n\"+\n     \"from bokeh.resources import INLINE\\n\"+\n     \"output_notebook(resources=INLINE)\\n\"+\n     \"\u003c/code\u003e\\n\"+\n     \"\u003c/div\u003e\"}};\n\n  function display_loaded() {\n    var el = document.getElementById(null);\n    if (el != null) {\n      el.textContent = \"BokehJS is loading...\";\n    }\n    if (root.Bokeh !== undefined) {\n      if (el != null) {\n        el.textContent = \"BokehJS \" + root.Bokeh.version + \" successfully loaded.\";\n      }\n    } else if (Date.now() \u003c root._bokeh_timeout) {\n      setTimeout(display_loaded, 100)\n    }\n  }\n\n\n  function run_callbacks() {\n    try {\n      root._bokeh_onload_callbacks.forEach(function(callback) {\n        if (callback != null)\n          callback();\n      });\n    } finally {\n      delete root._bokeh_onload_callbacks\n    }\n    console.debug(\"Bokeh: all callbacks have finished\");\n  }\n\n  function load_libs(css_urls, js_urls, callback) {\n    if (css_urls == null) css_urls = [];\n    if (js_urls == null) js_urls = [];\n\n    root._bokeh_onload_callbacks.push(callback);\n    if (root._bokeh_is_loading \u003e 0) {\n      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n      return null;\n    }\n    if (js_urls == null || js_urls.length === 0) {\n      run_callbacks();\n      return null;\n    }\n    console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n    root._bokeh_is_loading = css_urls.length + js_urls.length;\n\n    function on_load() {\n      root._bokeh_is_loading--;\n      if (root._bokeh_is_loading === 0) {\n        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n        run_callbacks()\n      }\n    }\n\n    function on_error(url) {\n      console.error(\"failed to load \" + url);\n    }\n\n    for (let i = 0; i \u003c css_urls.length; i++) {\n      const url = css_urls[i];\n      const element = document.createElement(\"link\");\n      element.onload = on_load;\n      element.onerror = on_error.bind(null, url);\n      element.rel = \"stylesheet\";\n      element.type = \"text/css\";\n      element.href = url;\n      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n      document.body.appendChild(element);\n    }\n\n    const hashes = {\"https://cdn.bokeh.org/bokeh/release/bokeh-2.3.3.min.js\": \"dM3QQsP+wXdHg42wTqW85BjZQdLNNIXqlPw/BgKoExPmTG7ZLML4EGqLMfqHT6ON\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.3.3.min.js\": \"8x57I4YuIfu8XyZfFo0XVr2WAT8EK4rh/uDe3wF7YuW2FNUSNEpJbsPaB1nJ2fz2\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.3.3.min.js\": \"3QTqdz9LyAm2i0sG5XTePsHec3UHWwVsrOL68SYRoAXsafvfAyqtQ+h440+qIBhS\"};\n\n    for (let i = 0; i \u003c js_urls.length; i++) {\n      const url = js_urls[i];\n      const element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error.bind(null, url);\n      element.async = false;\n      element.src = url;\n      if (url in hashes) {\n        element.crossOrigin = \"anonymous\";\n        element.integrity = \"sha384-\" + hashes[url];\n      }\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n  };\n\n  function inject_raw_css(css) {\n    const element = document.createElement(\"style\");\n    element.appendChild(document.createTextNode(css));\n    document.body.appendChild(element);\n  }\n\n  \n  var js_urls = [\"https://cdn.bokeh.org/bokeh/release/bokeh-2.3.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.3.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.3.3.min.js\"];\n  var css_urls = [];\n  \n\n  var inline_js = [\n    function(Bokeh) {\n      Bokeh.set_log_level(\"info\");\n    },\n    function(Bokeh) {\n    \n    \n    }\n  ];\n\n  function run_inline_js() {\n    \n    if (root.Bokeh !== undefined || force === true) {\n      \n    for (var i = 0; i \u003c inline_js.length; i++) {\n      inline_js[i].call(root, root.Bokeh);\n    }\n    } else if (Date.now() \u003c root._bokeh_timeout) {\n      setTimeout(run_inline_js, 100);\n    } else if (!root._bokeh_failed_load) {\n      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n      root._bokeh_failed_load = true;\n    } else if (force !== true) {\n      var cell = $(document.getElementById(null)).parents('.cell').data().cell;\n      cell.output_area.append_execute_result(NB_LOAD_WARNING)\n    }\n\n  }\n\n  if (root._bokeh_is_loading === 0) {\n    console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n    run_inline_js();\n  } else {\n    load_libs(css_urls, js_urls, function() {\n      console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n      run_inline_js();\n    });\n  }\n}(window));"
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "\n",
              "\n",
              "\n",
              "\n",
              "  \u003cdiv class=\"bk-root\" id=\"2f2d37ba-1716-4efd-ab9a-af3959c20f92\" data-root-id=\"1002\"\u003e\u003c/div\u003e\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/javascript": [
              "(function(root) {\n",
              "  function embed_document(root) {\n",
              "    \n",
              "  var docs_json = {\"f8e0cf83-1ec8-4efd-aa6f-6ea4ccc965c7\":{\"defs\":[],\"roots\":{\"references\":[{\"attributes\":{\"below\":[{\"id\":\"1013\"}],\"center\":[{\"id\":\"1016\"},{\"id\":\"1020\"},{\"id\":\"1051\"}],\"height\":400,\"left\":[{\"id\":\"1017\"}],\"output_backend\":\"webgl\",\"renderers\":[{\"id\":\"1039\"},{\"id\":\"1059\"},{\"id\":\"1080\"},{\"id\":\"1103\"},{\"id\":\"1128\"},{\"id\":\"1155\"},{\"id\":\"1184\"}],\"sizing_mode\":\"fixed\",\"title\":{\"id\":\"1003\"},\"toolbar\":{\"id\":\"1028\"},\"width\":1400,\"x_range\":{\"id\":\"1005\"},\"x_scale\":{\"id\":\"1009\"},\"y_range\":{\"id\":\"1007\"},\"y_scale\":{\"id\":\"1011\"}},\"id\":\"1002\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{\"line_color\":\"#2ca02c\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"fsim-y\"}},\"id\":\"1078\",\"type\":\"Line\"},{\"attributes\":{},\"id\":\"1025\",\"type\":\"ResetTool\"},{\"attributes\":{},\"id\":\"1026\",\"type\":\"HelpTool\"},{\"attributes\":{\"label\":{\"value\":\" butteraugli_6\"},\"renderers\":[{\"id\":\"1059\"}]},\"id\":\"1073\",\"type\":\"LegendItem\"},{\"attributes\":{\"source\":{\"id\":\"1055\"}},\"id\":\"1060\",\"type\":\"CDSView\"},{\"attributes\":{},\"id\":\"1044\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"1071\",\"type\":\"Selection\"},{\"attributes\":{},\"id\":\"1072\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1059\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"butteraugli_6\",\"@{butteraugli_6}\"]]},\"id\":\"1074\",\"type\":\"HoverTool\"},{\"attributes\":{},\"id\":\"1049\",\"type\":\"Selection\"},{\"attributes\":{\"source\":{\"id\":\"1035\"}},\"id\":\"1040\",\"type\":\"CDSView\"},{\"attributes\":{\"data_source\":{\"id\":\"1035\"},\"glyph\":{\"id\":\"1037\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1038\"},\"view\":{\"id\":\"1040\"}},\"id\":\"1039\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1042\",\"type\":\"AllLabels\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1094\"},\"selection_policy\":{\"id\":\"1095\"}},\"id\":\"1076\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#1f77b4\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"butteraugli_max\"}},\"id\":\"1038\",\"type\":\"Line\"},{\"attributes\":{\"label\":{\"value\":\" fsim-y\"},\"renderers\":[{\"id\":\"1080\"}]},\"id\":\"1096\",\"type\":\"LegendItem\"},{\"attributes\":{\"data_source\":{\"id\":\"1124\"},\"glyph\":{\"id\":\"1126\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1127\"},\"view\":{\"id\":\"1129\"}},\"id\":\"1128\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"click_policy\":\"hide\",\"items\":[{\"id\":\"1052\"},{\"id\":\"1073\"},{\"id\":\"1096\"},{\"id\":\"1121\"},{\"id\":\"1148\"},{\"id\":\"1177\"},{\"id\":\"1208\"}]},\"id\":\"1051\",\"type\":\"Legend\"},{\"attributes\":{\"bottom_units\":\"screen\",\"fill_alpha\":0.5,\"fill_color\":\"lightgrey\",\"left_units\":\"screen\",\"level\":\"overlay\",\"line_alpha\":1.0,\"line_color\":\"black\",\"line_dash\":[4,4],\"line_width\":2,\"right_units\":\"screen\",\"syncable\":false,\"top_units\":\"screen\"},\"id\":\"1027\",\"type\":\"BoxAnnotation\"},{\"attributes\":{\"source\":{\"id\":\"1076\"}},\"id\":\"1081\",\"type\":\"CDSView\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#2ca02c\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"fsim-y\"}},\"id\":\"1079\",\"type\":\"Line\"},{\"attributes\":{\"data_source\":{\"id\":\"1076\"},\"glyph\":{\"id\":\"1078\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1079\"},\"view\":{\"id\":\"1081\"}},\"id\":\"1080\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1094\",\"type\":\"Selection\"},{\"attributes\":{},\"id\":\"1005\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"1095\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1080\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"fsim-y\",\"@{fsim-y}\"]]},\"id\":\"1097\",\"type\":\"HoverTool\"},{\"attributes\":{\"line_color\":\"#d62728\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"fsim-rgb\"}},\"id\":\"1101\",\"type\":\"Line\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1119\"},\"selection_policy\":{\"id\":\"1120\"}},\"id\":\"1099\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"label\":{\"value\":\" fsim-rgb\"},\"renderers\":[{\"id\":\"1103\"}]},\"id\":\"1121\",\"type\":\"LegendItem\"},{\"attributes\":{},\"id\":\"1003\",\"type\":\"Title\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1049\"},\"selection_policy\":{\"id\":\"1050\"}},\"id\":\"1035\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#9467bd\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"msssim-y\"}},\"id\":\"1127\",\"type\":\"Line\"},{\"attributes\":{},\"id\":\"1047\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"1045\",\"type\":\"AllLabels\"},{\"attributes\":{\"source\":{\"id\":\"1099\"}},\"id\":\"1104\",\"type\":\"CDSView\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#d62728\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"fsim-rgb\"}},\"id\":\"1102\",\"type\":\"Line\"},{\"attributes\":{\"data_source\":{\"id\":\"1099\"},\"glyph\":{\"id\":\"1101\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1102\"},\"view\":{\"id\":\"1104\"}},\"id\":\"1103\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1050\",\"type\":\"UnionRenderers\"},{\"attributes\":{},\"id\":\"1119\",\"type\":\"Selection\"},{\"attributes\":{},\"id\":\"1120\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"label\":{\"value\":\" butteraugli_max\"},\"renderers\":[{\"id\":\"1039\"}]},\"id\":\"1052\",\"type\":\"LegendItem\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1103\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"fsim-rgb\",\"@{fsim-rgb}\"]]},\"id\":\"1122\",\"type\":\"HoverTool\"},{\"attributes\":{\"line_color\":\"#ff7f0e\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"butteraugli_6\"}},\"id\":\"1057\",\"type\":\"Line\"},{\"attributes\":{},\"id\":\"1007\",\"type\":\"DataRange1d\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1071\"},\"selection_policy\":{\"id\":\"1072\"}},\"id\":\"1055\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"line_color\":\"#9467bd\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"msssim-y\"}},\"id\":\"1126\",\"type\":\"Line\"},{\"attributes\":{\"label\":{\"value\":\" msssim-y\"},\"renderers\":[{\"id\":\"1128\"}]},\"id\":\"1148\",\"type\":\"LegendItem\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1146\"},\"selection_policy\":{\"id\":\"1147\"}},\"id\":\"1124\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"data_source\":{\"id\":\"1055\"},\"glyph\":{\"id\":\"1057\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1058\"},\"view\":{\"id\":\"1060\"}},\"id\":\"1059\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#ff7f0e\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"butteraugli_6\"}},\"id\":\"1058\",\"type\":\"Line\"},{\"attributes\":{\"source\":{\"id\":\"1124\"}},\"id\":\"1129\",\"type\":\"CDSView\"},{\"attributes\":{},\"id\":\"1147\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"axis\":{\"id\":\"1013\"},\"ticker\":null},\"id\":\"1016\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"1146\",\"type\":\"Selection\"},{\"attributes\":{\"formatter\":{\"id\":\"1044\"},\"major_label_policy\":{\"id\":\"1042\"},\"ticker\":{\"id\":\"1018\"}},\"id\":\"1017\",\"type\":\"LinearAxis\"},{\"attributes\":{\"line_color\":\"#8c564b\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"nlpd-y\"}},\"id\":\"1153\",\"type\":\"Line\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1128\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"msssim-y\",\"@{msssim-y}\"]]},\"id\":\"1149\",\"type\":\"HoverTool\"},{\"attributes\":{},\"id\":\"1018\",\"type\":\"BasicTicker\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1175\"},\"selection_policy\":{\"id\":\"1176\"}},\"id\":\"1151\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"axis\":{\"id\":\"1017\"},\"dimension\":1,\"ticker\":null},\"id\":\"1020\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"1176\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"label\":{\"value\":\" nlpd-y\"},\"renderers\":[{\"id\":\"1155\"}]},\"id\":\"1177\",\"type\":\"LegendItem\"},{\"attributes\":{\"line_color\":\"#1f77b4\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"butteraugli_max\"}},\"id\":\"1037\",\"type\":\"Line\"},{\"attributes\":{\"source\":{\"id\":\"1151\"}},\"id\":\"1156\",\"type\":\"CDSView\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#8c564b\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"nlpd-y\"}},\"id\":\"1154\",\"type\":\"Line\"},{\"attributes\":{\"data_source\":{\"id\":\"1151\"},\"glyph\":{\"id\":\"1153\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1154\"},\"view\":{\"id\":\"1156\"}},\"id\":\"1155\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1175\",\"type\":\"Selection\"},{\"attributes\":{\"line_color\":\"#e377c2\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"ssimulacra\"}},\"id\":\"1182\",\"type\":\"Line\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1039\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"butteraugli_max\",\"@{butteraugli_max}\"]]},\"id\":\"1053\",\"type\":\"HoverTool\"},{\"attributes\":{\"active_multi\":null,\"active_scroll\":{\"id\":\"1022\"},\"tools\":[{\"id\":\"1021\"},{\"id\":\"1022\"},{\"id\":\"1023\"},{\"id\":\"1024\"},{\"id\":\"1025\"},{\"id\":\"1026\"},{\"id\":\"1053\"},{\"id\":\"1074\"},{\"id\":\"1097\"},{\"id\":\"1122\"},{\"id\":\"1149\"},{\"id\":\"1178\"},{\"id\":\"1209\"}]},\"id\":\"1028\",\"type\":\"Toolbar\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1155\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"nlpd-y\",\"@{nlpd-y}\"]]},\"id\":\"1178\",\"type\":\"HoverTool\"},{\"attributes\":{},\"id\":\"1014\",\"type\":\"BasicTicker\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1206\"},\"selection_policy\":{\"id\":\"1207\"}},\"id\":\"1180\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"label\":{\"value\":\" ssimulacra\"},\"renderers\":[{\"id\":\"1184\"}]},\"id\":\"1208\",\"type\":\"LegendItem\"},{\"attributes\":{},\"id\":\"1021\",\"type\":\"PanTool\"},{\"attributes\":{\"source\":{\"id\":\"1180\"}},\"id\":\"1185\",\"type\":\"CDSView\"},{\"attributes\":{\"axis_label\":\"elo\",\"formatter\":{\"id\":\"1047\"},\"major_label_policy\":{\"id\":\"1045\"},\"ticker\":{\"id\":\"1014\"}},\"id\":\"1013\",\"type\":\"LinearAxis\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#e377c2\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"ssimulacra\"}},\"id\":\"1183\",\"type\":\"Line\"},{\"attributes\":{\"data_source\":{\"id\":\"1180\"},\"glyph\":{\"id\":\"1182\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1183\"},\"view\":{\"id\":\"1185\"}},\"id\":\"1184\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1009\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"1206\",\"type\":\"Selection\"},{\"attributes\":{},\"id\":\"1011\",\"type\":\"LinearScale\"},{\"attributes\":{\"overlay\":{\"id\":\"1027\"}},\"id\":\"1023\",\"type\":\"BoxZoomTool\"},{\"attributes\":{},\"id\":\"1207\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1184\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"ssimulacra\",\"@{ssimulacra}\"]]},\"id\":\"1209\",\"type\":\"HoverTool\"},{\"attributes\":{},\"id\":\"1024\",\"type\":\"SaveTool\"},{\"attributes\":{},\"id\":\"1022\",\"type\":\"WheelZoomTool\"}],\"root_ids\":[\"1002\"]},\"title\":\"Bokeh Application\",\"version\":\"2.3.3\"}};\n",
              "  var render_items = [{\"docid\":\"f8e0cf83-1ec8-4efd-aa6f-6ea4ccc965c7\",\"root_ids\":[\"1002\"],\"roots\":{\"1002\":\"2f2d37ba-1716-4efd-ab9a-af3959c20f92\"}}];\n",
              "  root.Bokeh.embed.embed_items_notebook(docs_json, render_items);\n",
              "\n",
              "  }\n",
              "  if (root.Bokeh !== undefined) {\n",
              "    embed_document(root);\n",
              "  } else {\n",
              "    var attempts = 0;\n",
              "    var timer = setInterval(function(root) {\n",
              "      if (root.Bokeh !== undefined) {\n",
              "        clearInterval(timer);\n",
              "        embed_document(root);\n",
              "      } else {\n",
              "        attempts++;\n",
              "        if (attempts \u003e 100) {\n",
              "          clearInterval(timer);\n",
              "          console.log(\"Bokeh: ERROR: Unable to run BokehJS code because BokehJS library is missing\");\n",
              "        }\n",
              "      }\n",
              "    }, 10, root)\n",
              "  }\n",
              "})(window);"
            ],
            "application/vnd.bokehjs_exec.v0+json": ""
          },
          "metadata": {
            "application/vnd.bokehjs_exec.v0+json": {
              "id": "1002"
            }
          },
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv style=\"display: table;\"\u003e\u003cdiv style=\"display: table-row;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003cb title=\"bokeh.plotting.figure.Figure\"\u003eFigure\u003c/b\u003e(\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eid\u0026nbsp;=\u0026nbsp;'1002', \u003cspan id=\"1344\" style=\"cursor: pointer;\"\u003e\u0026hellip;)\u003c/span\u003e\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eabove\u0026nbsp;=\u0026nbsp;[],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ealign\u0026nbsp;=\u0026nbsp;'start',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003easpect_ratio\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003easpect_scale\u0026nbsp;=\u0026nbsp;1,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ebackground\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ebackground_fill_alpha\u0026nbsp;=\u0026nbsp;1.0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ebackground_fill_color\u0026nbsp;=\u0026nbsp;'#ffffff',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ebelow\u0026nbsp;=\u0026nbsp;[LinearAxis(id='1013', ...)],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eborder_fill_alpha\u0026nbsp;=\u0026nbsp;1.0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eborder_fill_color\u0026nbsp;=\u0026nbsp;'#ffffff',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ecenter\u0026nbsp;=\u0026nbsp;[Grid(id='1016', ...), Grid(id='1020', ...), Legend(id='1051', ...)],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ecss_classes\u0026nbsp;=\u0026nbsp;[],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003edisabled\u0026nbsp;=\u0026nbsp;False,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eextra_x_ranges\u0026nbsp;=\u0026nbsp;{},\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eextra_y_ranges\u0026nbsp;=\u0026nbsp;{},\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eframe_height\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eframe_width\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eheight\u0026nbsp;=\u0026nbsp;400,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eheight_policy\u0026nbsp;=\u0026nbsp;'auto',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ehidpi\u0026nbsp;=\u0026nbsp;True,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003einner_height\u0026nbsp;=\u0026nbsp;0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003einner_width\u0026nbsp;=\u0026nbsp;0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ejs_event_callbacks\u0026nbsp;=\u0026nbsp;{},\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ejs_property_callbacks\u0026nbsp;=\u0026nbsp;{},\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eleft\u0026nbsp;=\u0026nbsp;[LinearAxis(id='1017', ...)],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003elod_factor\u0026nbsp;=\u0026nbsp;10,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003elod_interval\u0026nbsp;=\u0026nbsp;300,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003elod_threshold\u0026nbsp;=\u0026nbsp;2000,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003elod_timeout\u0026nbsp;=\u0026nbsp;500,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emargin\u0026nbsp;=\u0026nbsp;(0, 0, 0, 0),\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ematch_aspect\u0026nbsp;=\u0026nbsp;False,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emax_height\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emax_width\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_border\u0026nbsp;=\u0026nbsp;5,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_border_bottom\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_border_left\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_border_right\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_border_top\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_height\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_width\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ename\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eouter_height\u0026nbsp;=\u0026nbsp;0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eouter_width\u0026nbsp;=\u0026nbsp;0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_alpha\u0026nbsp;=\u0026nbsp;1.0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_cap\u0026nbsp;=\u0026nbsp;'butt',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_color\u0026nbsp;=\u0026nbsp;'#e5e5e5',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_dash\u0026nbsp;=\u0026nbsp;[],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_dash_offset\u0026nbsp;=\u0026nbsp;0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_join\u0026nbsp;=\u0026nbsp;'bevel',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_width\u0026nbsp;=\u0026nbsp;1,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutput_backend\u0026nbsp;=\u0026nbsp;'webgl',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003erenderers\u0026nbsp;=\u0026nbsp;[GlyphRenderer(id='1039', ...), GlyphRenderer(id='1059', ...), GlyphRenderer(id='1080', ...), GlyphRenderer(id='1103', ...), GlyphRenderer(id='1128', ...), GlyphRenderer(id='1155', ...), GlyphRenderer(id='1184', ...)],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ereset_policy\u0026nbsp;=\u0026nbsp;'standard',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eright\u0026nbsp;=\u0026nbsp;[],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003esizing_mode\u0026nbsp;=\u0026nbsp;'fixed',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003esubscribed_events\u0026nbsp;=\u0026nbsp;[],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003esyncable\u0026nbsp;=\u0026nbsp;True,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003etags\u0026nbsp;=\u0026nbsp;[],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003etitle\u0026nbsp;=\u0026nbsp;Title(id='1003', ...),\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003etitle_location\u0026nbsp;=\u0026nbsp;'above',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003etoolbar\u0026nbsp;=\u0026nbsp;Toolbar(id='1028', ...),\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003etoolbar_location\u0026nbsp;=\u0026nbsp;'right',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003etoolbar_sticky\u0026nbsp;=\u0026nbsp;True,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003evisible\u0026nbsp;=\u0026nbsp;True,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ewidth\u0026nbsp;=\u0026nbsp;1400,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ewidth_policy\u0026nbsp;=\u0026nbsp;'auto',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ex_range\u0026nbsp;=\u0026nbsp;DataRange1d(id='1005', ...),\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ex_scale\u0026nbsp;=\u0026nbsp;LinearScale(id='1009', ...),\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ey_range\u0026nbsp;=\u0026nbsp;DataRange1d(id='1007', ...),\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ey_scale\u0026nbsp;=\u0026nbsp;LinearScale(id='1011', ...))\u003c/div\u003e\u003c/div\u003e\u003c/div\u003e\n",
              "\u003cscript\u003e\n",
              "(function() {\n",
              "  var expanded = false;\n",
              "  var ellipsis = document.getElementById(\"1344\");\n",
              "  ellipsis.addEventListener(\"click\", function() {\n",
              "    var rows = document.getElementsByClassName(\"1343\");\n",
              "    for (var i = 0; i \u003c rows.length; i++) {\n",
              "      var el = rows[i];\n",
              "      el.style.display = expanded ? \"none\" : \"table-row\";\n",
              "    }\n",
              "    ellipsis.innerHTML = expanded ? \"\u0026hellip;)\" : \"\u0026lsaquo;\u0026lsaquo;\u0026lsaquo;\";\n",
              "    expanded = !expanded;\n",
              "  });\n",
              "})();\n",
              "\u003c/script\u003e\n"
            ],
            "text/plain": [
              "Figure(id='1002', ...)"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "rollingcorr(metrics, 'spearman', 5000, 1000).plot_bokeh(x='elo', figsize=(1400, 400))"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "correlations.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
